Adequacy and Evaluation

Currently, integration of planning and generation is an active research area. New insights are appearing from research on speakers’ intentions and attitudes, tactical planning, argumentation strategies, dialogue management, and system design. In this paper I will explore the new directions under two general headings: conversational adequacy and system evaluation. 1 Conversational adequacy Fluency of communication is dependent on the inferences that the participants are able to draw, and on adequate reporting of the results of this evaluation. However, what are the correct inferences and appropriate responses like? Why would a particular response be favoured over another? There is not much concensus over what is a conversationally adequate response or how to determine one, except that the requirements should mirror some standard of appropriate behaviour in a given situation, and adequacy of system responses should be defined with respect to dialogues that naturally occur in that situation. Using a corpus of typical dialogues that the system is to handle provides an empirical foundation for the adequacy of responses, although in the actual system the responses are usually modified to accommodate a limited linguistic repertoire. Consider the following sample dialogue in an information seeking context: (1) User1: Is there anywhere in the town centre that serves hot, spicy food? Wizard1: What type of food? User2: Hot, spicy food. Far eastern or Asian, perhaps. Given that the wizard’s task is to provide the user with information about restaurants, but the user specification hot, spicy food is not clear enough to pick out a restaurant type, the response sounds natural. However, for a system, the generation of such a response is not that easy. First the system has to recognize the metonymic relation between the concepts ‘food’ and ‘restaurant’ (if the user is looking for a place that serves food, this is taken to be a restaurant, as opposed to other places associated with food like take-aways and shops), then formulate its own goal to restrict the database search with the help of the restaurant type (it would be uncooperative to give the user information about all the restaurants), relate the modifiers hot and spicy to a restaurant ’type’, and finally, generate an elliptical response using the meta-concept type and the word food used in the user’s contribution. A system might thus formulate a slightly different response like What type of restaurant? or I don’t know ’hot, spicy food’. Although 1 Nara Institute of Science and Technology, 8916-5 Takayama, Ikoma, Nara, 630-01 JAPAN. Email: [email protected] these responses may not be as natural as the one in example (1) (abrupt topic shift from food to restaurant, and less cooperative statement of one’s knowledge limits), they nevertheless convey relevant information to the user. The former explicitly signals what information is looked for, thus allowing the user to correct the interpretation if the connection between her request and the application model is wrong. The latter prevents the user from repeating the unknown adjectives in her response, and thus helps the wizard to proceed with the task without digressing into a repetition of the restaurant type question. Assessing the success of a contribution in a particular dialogue situation is not a simple task. The first criterion for an appropriate response seems to lie in the general aims of the dialogue model: in human-computer interaction the latter responses would be fine, justified by the requirements of efficiency and feasibility, and by the assumption that humans are still far better than computers in bridging inference gaps. In analogous human-human dialogues, say those at a Tourist Information Office, these responses or ones like I don’t understand what you mean, No, I do not know, or repetitions of the same question etc., usually reinforce an impression of a rude and uncooperative partner, and the fluency of communication suffers seriously. On the other hand, modelling techniques should be developed simultaneously with linguistic capabilities, and with appropriate user interfaces. As noted e.g. by Cawsey and Grasso (these proceedings), existing simple techniques, though overlooking the complexity of the domain and the reasoning involved, may provide a safer basis for application systems than a fancier yet unreliable sophistication of communicative capabilities. Besides the external system design viewpoint, example (1) also exemplifies the notion of communicative strategy: the way in which mutual knowledge is established, maintained, modified and exploited. Depending on the speaker’s conversational posture and risk-taking ability ([5]), some part of the information content can be ignored or omitted in order to contribute to conversational fluency, to achieve a particular communicative effect, to maintain one’s ’face’, etc. Moreover, an appropriate way to react does not require truthfulness in the sense that the response details one’s knowledge limits. For instance, the wizard does not explicitly tell whether hot and spicy are truly understood, or that she assumes the user is looking for a restaurant rather than a fast food take-away or an exotic shop only because she can find no information about the latter two in the database. On the contrary, the response strongly implies that the wizard understands the meaning of the modifiers and can associate ’serving food’ with ’restaurants’, and only requires some clarification of the type of food. The cooperative partner chooses to specify the food type further, the wizard continues with the task, and the user gets a list of restaurants 2 After an open question like Wizard1, the user could use the same unknown adjectives in her reply and thus give no useful information at all. In the example, the user indeed repeats the adjectives, although she gives further specifications as well. c 1996 K. Jokinen Proceedings of the ECAI 96 Workshop Gaps and Bridges: New Directions in Planning and Natural Language Generation. Edited by K. Jokinen, M. Maybury, M. Zock and I.Zukerman. probably without even noticing that there was something vague in her first request. The situation becomes more complicated if the user chooses to elaborate the unknown concepts with other unknown concepts: Well, hot in taste, not in temperature; spicy as being highly flavoured. Backtracking is costly not only computationally but also in terms of time and money, if the user is connected to an online information service. The effectiveness of a communicative strategy is thus a tradeoff between cost and benefit: an efficient although not too detailed response may cause misunderstandings and lengthy explanations later, thus adding to the overall cost of the dialogue in a global sense, while a less risky strategy may overlook safe inferences which would make the communication more efficient in a local sense. The success of a strategy also depends on the partner and her cooperation. For instance, the crucial point in User2 is that the user specifies the unclear modifiers with more appropriate ones, instead of e.g. questioning the relevance of the question (What type of food? Hot and spicy as I said). This brings us to another point mentioned in the workshop call: cooperation and conflict resolution. Communication is an interactive process, and successful communication requires that the participants cooperate at least in the general task of information transfer: they have a common goal, they show consideration to each other’s goals, and they trust that the partner behaves according to the same cooperative principles ([2]). Hence, cooperation is not a means to guarantee comprehensibility of individual utterances, but an effect which emerges from the dialogue partners’ willingness to continue the dialogue, and their ability to make relevant contributions until both are satisfied as having achieved their goals. However, cooperation does not mean that agents always try to react in the way the partner intends to evoke. As [8] points out, if agents are always in agreement and ready to adopt the other’s goals, they are benevolent rather than cooperative. Autonomous agents also find themselves in conflict situations and communication is the primary means to resolve them. To model the control that an agent has over decisions concerning which goal to pursue, some measurement of the agent’s preferences and autonomy are required besides reasoning about intentions. Within game theory, cooperation has been investigated in regard to optimal strategies that autonomous agents, seeking their own benefit in interaction with other agents, should use to get desired results in different social dilemma situations. Using the Prisoner’s Dilemma game, it has been shown that cooperation emerges from the expectation for repeated interactions, and that a "tit for tat" strategy which mimics the partner’s play in her latest game is effective for such iterative situations ([3]). Conversational adequacy manifests itself in the contributions that the agent plans to clear up vagueness, misunderstandings or lack of understanding in the partner’s contributions. Planning an appropriate response thus actually starts from the interpretation of the previous contribution: given that the agent is pushed onto a wrong track by a deficient or non-intended interpretation, her reaction will also be less adequate than it would have been otherwise. However, a resourcebounded agent cannot be certain that her interpretation of a particular contribution is the correct one: the information available for decisions need be neither complete nor correct. Response planning must thus accommodate uncertainty and produce a contribution on the basis of facts which may turn out to be false, and inferences which may be invalid when more information becomes available. In dialogue situations, uncertainty is dealt with by the flexibility of the medium itself: negotiating, giving feedback and explanations. A common ground is gradually built over a set of contributions (cf. [7]) rather than in a ’one-shot’ process where all the information is given to the partner at once ([13], cf. also Inui et al., these proceedings). If there are any dialogue plans, they must be general and partial so that they can be modified, specified and augmented according to the on-going dialogue. However, I would claim that dialogue planning is a reactive rather than a deliberative process, and that communicative principles, effectively implementing the agent’s rationality, sincerity, motivation and consideration, play a crucial role in reasoning about appropriate continuations. The underlying task gives rise to dialogue acts (e.g. one wants to know about restaurants in order to plan a friend’s visit) and also controls what must be taken into consideration and in which order (to look for a restaurant that serves hot, spicy food, one must first know that the friend likes this kind of food; and after getting a list of suitable restaurants, one may want to check if they are easily accessible by public transportation, etc.), but the reaction itself is based on contextual changes resulting from the evaluation of the partner’s response, rather than on a realisation of a predefined structure of possible dialogue moves ([11]).